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[Title of Document] SPECIFICATION 

[Title of the Invention] Image Processing System, Projector, Portable Device, and 

Image Processing Method 

[Claims] 

5 [Claim 1] An image processing system comprising: 

sensing means for sensing a projection area on which an image is projected and 
outputting sensing information; 

histogram generation means for generating histogram information that expresses 
histograms of the numbers of pixels of the sensed image in each of the vertical and 
l o horizontal directions, based on the sensing information; 

direction determination means for generating direction information, based on an 
angle between a normal direction orthogonal to the projection area from a sensing 
position and a projected image direction toward a central portion of the projected image 
from the sensing position; and 
15 keystone distortion correction means for correcting keystone distortion of the 

projected image, based on the histogram information and the direction information. 

[Claim 2] The image processing system as defined in claim 1, comprising: 
a portable device having the sensing means, the histogram means, and the 
20 direction determination means; and 

an image display device having the keystone distortion correction means and 
projection means for projecting an image, 

wherein the keystone distortion correction means includes: 

vertical keystone distortion correction means having inclination detection means 
25 for detecting the inclination of the image display device, for correcting keystone 
distortion in the vertical direction; and 
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horizontal keystone distortion correction means for correcting keystone 
distortion in the horizontal direction, based on the histogram information and the 
direction information. 

5 [Claim 3] An image processing system comprising: 

light projection means for projecting light of a distortion-free and predetermined 
shape into a projection area; 

sensing means for outputting sensing information obtained by sensing the 
projection area into which the light of the predetermined shape is projected, from a 
10 projection position of a projection section for projecting an image; 

area extraction means for extracting coordinates of the light of the 
predetermined shape in a sensed area, based on the sensing information; and 

keystone distortion correction means for converting the coordinates of the light 
of the predetermined shape in the sensed area into coordinates for projection in a spatial 
,5 light modulator of the projection section, and correcting keystone distortion by mapping 
coordinates of an input image into the projection coordinates. 

[Claim 4] The image processing system as defined in claim 3, wherein: 
the predetermined shape is rectangular or square; 
20 the area extraction means extracts coordinates of four corners of the light of the 

predetermined shape; and 

the keystone distortion correction means converts the coordinates of the four 
corners of the light of the predetermined shape in the sensed area into coordinates for 
projection in a spatial light modulator of the projection section, and corrects keystone 
25 distortion by mapping the coordinates of an input image into an area formed by the 
projection coordinates of the four corners. 
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[Claim 5] The image processing system as defined in claim 3 or 4, comprising: 
a portable device having the light projection means; and 

an image display device having the sensing means, the keystone distortion 
correction means, and the projection section. 

[Claim 6] A projector comprising: 
projection means for projecting an image; 

receiving means for receiving histogram information that expresses histograms 
of the numbers of pixels of the sensed image in each of the vertical and horizontal 
directions, based on sensing information obtained by sensing a projection area on which 
the projected image is projected, and direction information based on an angle between a 
normal direction orthogonal to a projection area from a sensing position and a projected 
image direction toward a central portion of the projected image from the sensing 
position: and 

keystone distortion correction means for correcting keystone distortion of the 
projected image, based on the histogram information and the direction information. 

[Claim 7] The projector as defined in claim 6, 

wherein the keystone distortion correction means includes: 

vertical keystone distortion correction means having inclination detection means 
for detecting the inclination of the projection means, for correcting keystone distortion 
in the vertical direction; and 

horizontal keystone distortion correction means for correcting keystone 
distortion in the horizontal direction, based on the histogram information and the 
direction information. 



[Claim 8] A projector comprising: 
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sensing means for outputting sensing information obtained by sensing a 
projection area into which light of a distortion-free and predetermined shape is projected, 
from a projection position of a projection section for projecting an image; 

area extraction means for extracting coordinates of the light of the 
predetermined shape in a sensed area, based on the sensing information; 

keystone distortion correction means for converting the coordinates of the light 
of the predetermined shape in the sensed area into coordinates for projection in a spatial 
light modulator of the projection section, and correcting keystone distortion by mapping 
coordinates of an input image into the projection coordinates; and 

the projection section. 

[Claim 9] The projector as defined in claim 8, wherein: 
the predetermined shape is rectangular or square; 

the area extraction means extracts coordinates of four corners of the light of the 
predetermined shape; and 

the keystone distortion correction means converts the coordinates of the four 
corners of the light of the predetermined shape in the sensed area into coordinates for 
projection in a spatial light modulator of the projection section, and corrects keystone 
distortion by mapping the coordinates of an input image into an area formed by the 
projection coordinates of the four corners. 

[Claim 1 0] A portable device comprising: 

sensing means for sensing a projection area on which an image is projected; 

histogram generation means for generating histogram information that expresses 
histograms of the numbers of pixels of the sensed image in each of the vertical and 
horizontal directions, based on sensing information obtained by the sensing means; 



direction determination means for determining direction information based on an 
angle between a normal direction orthogonal to the projection area from a sensing 
position and a projected image direction toward a central portion of the projected image 
from the sensing position; and 
5 transmitter means for correcting keystone distortion of the projected image, 

based on the histogram information and the direction information, and transmitting the 
histogram information and the direction information to an image display device that 
projects the projected image. 

10 [Claim 1 1 ] An image processing method comprising: 

sensing a projection area on which an image is projected andoutputting sensing 
information; 

generating histogram information that expresses histograms of the numbers of 
pixels of the sensed image in each of the vertical and horizontal directions, based on the 
15 sensing information; 

detecting a normal direction orthogonal to the projection area from a sensing 

position; 

detecting a projected image direction toward a central portion of the projected 
image from the sensing position; 

generating direction information, based on an angle between the normal 
direction and the projected image direction; and 

correcting keystone distortion of the projected image, based on the histogram 
information and the direction information. 

[Claim 12] The image processing method as defined in claim 1 1 , comprising: 
detecting inclination of an image display device; 
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correcting keystone distortion in the vertical direction, based on the detected 
inclination; and 

correcting keystone distortion in the horizontal direction, based on the histogram 
information and the direction information. 



[Claim 13] An image processing method comprising: 

projecting light of a distortion-free and predetermined shape into a projection 

area; 

outputting sensing information obtained by sensing the projection area into 
which the light of the predetermined shape is projected, from a projection position of a 
projection section for projecting an image; 

extracting coordinates of the light of the predetermined shape in a sensed area, 
based on the sensing information; and 

converting the coordinates of the light of the predetermined shape in the sensed 
area into coordinates for projection in a spatial light modulator of the projection section, 
and correcting keystone distortion by mapping coordinates of an input image into the in 
the sensed area projection coordinates. 

[Claim 14] The image processing method as defined in claim 13, 
wherein the predetermined shape is rectangular or square; and the method 
including: 

extracting coordinates of four corners of the light of the predetermined shape 
when coordinates of the light of the predetermined shape in the sensed area are 
extracted; and 

converting the coordinates of the four corners of the light of the predetermined 
shape in the sensed area into coordinates for projection in the spatial light modulator of 
the projection section, and correcting keystone distortion by mapping coordinates of an 
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input image into an area formed by the projection coordinates of the four corners, when 
the keystone distortion is corrected. 

[Detailed Description of the Invention] 
[0001] 

[Technical Field of the Invention] 

The present invention relates to an image processing system, a projector, a 
portable device, and an image processing method that are capable of correcting 
keystone distortion. 

[0002] 

[Background of the Invention] 

Various different methods have been proposed for correcting keystone distortion 
that is generated in images projected by a projector. 
[0003] 

One method detects the inclination of the projector and automatically corrects 
keystone distortion in the vertical direction, by way of example. 
[0004] 

However, this method does not enable the user to correct keystone distortion in 
the horizontal direction. 
[0005] 

For that reason, one method of correcting keystone distortion in the horizontal 
direction involves the user correcting the keystone distortion manually by pressing a 
correction switch on a remote controller while viewing the image. 

[0006] 

However, the manual correction of the keystone distortion is troublesome to the 
user, making it difficult to perform suitable correction manually. 
[0007] 
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In the light of the above problem, methods have been proposed for correcting 
image distortion automatically, such as that disclosed in a patent literature 1 . 
[0008] 

[Patent Literature 1] 

Japanese Patent Publication No. 2002-1 12148 

[0009] 

[Problems to be Solved by the Invention] 

However, this method only discloses a method of correcting keystone distortion 
by detecting the distance up to the installation plane of the projector and the inclinations 
in the horizontal and vertical directions with respect to the horizontal plane of the 
projector. 

[0010] 

For that reason, that method is based on the premise that the projection screen is 
positioned facing the projector, as stated in that application. 
[0011] 

If the projector is positioned in a direction at an angle to the projection screen, 
however, keystone distortion is generated in the horizontal direction (sideways 
direction). 

[0012] 

It is therefore difficult to use that published method to correct keystone 
distortion in the horizontal direction both automatically and appropriately. 
[0013] 

The present invention was devised in the light of the above-described technical 
problems, and may provide an image processing system, a projector, a portable device, 
and an image processing method that make it possible to correct keystone distortion, 
particularly keystone distortion in the horizontal direction, both automatically and 
appropriately. 



[0014] 

[Means for Solving Problems] 

In order to resolve the above-described technical problems, the present invention 
relates to an image processing system comprising: 
5 sensing means for sensing a projection area on which an image is projected and 

outputting sensing information; 

histogram generation means for generating histogram information that expresses 
histograms of the numbers of pixels of the sensed image in each of the vertical and 
horizontal directions, based on the sensing information; 
10 direction determination means for generating direction information, based on an 

angle between a normal direction orthogonal to the projection area from a sensing 
position and a projected image direction toward a central portion of the projected image 
from the sensing position; and 

keystone distortion correction means for correcting keystone distortion of the 
15 projected image, based on the histogram information and the direction information. 
[0015] 

The present invention relates to a projector comprising: 
projection means for projecting an image; 

receiving means for receiving histogram information that expresses histograms 
20 of the numbers of pixels of the sensed image in each of the vertical and horizontal 
directions, based on sensing information obtained by sensing a projection area on which 
the projected image is projected, and direction information based on an angle between a 
normal direction orthogonal to a projection area from a sensing position and a projected 
image direction toward a central portion of the projected image from the sensing 
25 position; and 

keystone distortion correction means for correcting keystone distortion of the 
projected image, based on the histogram information and the direction information. 
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[0016] 

The present invention relates to a portable device comprising: 
sensing means for sensing a projection area on which an image is projected; 
histogram generation means for generating histogram information that expresses 
histograms of the numbers of pixels of the sensed image in each of the vertical and 
horizontal directions, based on sensing information obtained by the sensing means: 

direction determination means for determining direction information based on an 
angle between a normal direction orthogonal to the projection area from a sensing 
position and a projected image direction toward a central portion of the projected image 
from the sensing position: and 

transmitter means for correcting keystone distortion of the projected image, 
based on the histogram information and the direction information, and transmitting the 
histogram information and the direction information to an image display device that 
projects the projected image. 
[0017] 

The present invention relates to an image processing method comprising: 
sensing a projection area on which an image is projected and outputting sensing 
information: 

generating histogram information that expresses histograms of the numbers of 
pixels of the sensed image in each of the vertical and horizontal directions, based on the 
sensing information: 

detecting a normal direction orthogonal to the projection area from a sensing 

position: 

detecting a projected image direction toward a central portion of the projected 
image from the sensing position; 

generating direction information, based on an angle between the normal 
direction and the projected image direction: and 
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correcting keystone distortion of the projected image, based on the histogram 
information and the direction information. 
[0018] 

The image processing system and others according to the present invention 
makes it possible to determine keystone distortion by sensing. They can convert the 
magnitude of the distortion into numerical values by converting the sensing information 
into histograms. They can also determine the relative displacement between the sensing 
direction and the projection direction by determining the angle. 

[0019] 

Furthermore, the image processing system and others can correct not only 
keystone distortion in the vertical direction but also keystone distortion in the horizontal 
direction, both automatically and appropriately, by correcting the keystone distortion, 
based on histogram information that expresses the distortion and direction information 
that expresses relative displacement between the sensing direction and the projection 
direction. 

[0020] 

The image processing system may comprise: 

a portable device having the sensing means, the histogram means, and the 
direction determination means; and 

an image display device having the keystone distortion correction means and 
projection means for projecting an image, 

wherein the keystone distortion correction means may include: 

vertical keystone distortion correction means having inclination detection means 
for detecting the inclination of the image display device, for correcting keystone 
> distortion in the vertical direction: and 
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horizontal keystone distortion correction means for correcting keystone 
distortion in the horizontal direction, based on the histogram information and the 
direction information. 

[0021] 

In the projector, the keystone distortion correction means may include: 

vertical keystone distortion correction means having inclination detection means 

for detecting the inclination of the projection means, for correcting keystone distortion 

in the vertical direction; and 

horizontal keystone distortion correction means for correcting keystone 

distortion in the horizontal direction, based on the histogram information and the 

direction information. 
[0022] 

The image processing method may comprise: 
detecting inclination of an image display device: 

correcting keystone distortion in the vertical direction, based on the detected 
inclination; and 

correcting keystone distortion in the horizontal direction, based on the histogram 
information and the direction information. 
[0023] 

The keystone distortion in the horizontal direction is affected by the relative 
positional relationship between the image display device and the projection area. In 
contrast thereto, the keystone distortion in the vertical direction is mainly affected by 
the inclination of the image display device. 

[0024] 

S The image processing system and others can therefore correct the keystone 

distortion as appropriate, by correcting the keystone distortion in the vertical direction, 
based on the inclination of the image display device, and by correcting the keystone 
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distortion in the horizontal direction, based on histogram information and direction 
information that reflect the position of the projection area. 
[0025] 

The present invention relates to an image processing system comprising: 

light projection means for. projecting light of a distortion-free and predetermined 
shape into a projection area: 

sensing means for outputting sensing information obtained by sensing the 
projection area into which the light of the predetermined shape is projected, from a 
projection position of a projection section for projecting an image: 

area extraction means for extracting coordinates of the light of the 
predetermined shape in a sensed area, based on the sensing information; and 

keystone distortion correction means for converting the coordinates of the light 
of the predetermined shape in the sensed area into coordinates for projection in a spatial 
light modulator of the projection section, and correcting keystone distortion by mapping 
coordinates of an input image into the projection coordinates. 
[0026] 

The image processing system may comprise: 

a portable device having the light projection means; and 

an image display device having the sensing means, the keystone distortion 
correction means, and the projection section. 
[0027] 

The present invention relates to a projector comprising: 

sensing means for outputting sensing information obtained by sensing a 
projection area into which light of a distortion-free and predetermined shape is projected, 
from a projection position of a projection section for projecting an image; 

area extraction means for extracting coordinates of the light of the 
predetermined shape in a sensed area, based on the sensing information: 
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keystone distortion correction means for converting the coordinates of the light 
of the predetermined shape in the sensed area into coordinates for projection in a spatial 
light modulator of the projection section, and correcting keystone distortion by mapping 
coordinates of an input image into the projection coordinates; and 

the projection section. 

[0028] 

The present invention relates to an image processing method comprising: 
projecting light of a distortion-free and predetermined shape into a projection 

area; 

outputting sensing information obtained by sensing the projection area into 
which the light of the predetermined shape is projected, from a projection position of a 
projection section for projecting an image; 

extracting coordinates of the light of the predetermined shape in a sensed area, 
based on the sensing information; and 

converting the coordinates of the light of the predetermined shape in the sensed 
area into coordinates for projection in a spatial light modulator of the projection sectjon, 
and correcting keystone distortion by mapping coordinates of an input image into the in 
the sensed area projection coordinates. 

[0029] 

The image processing system and others according to the present invention 
makes it possible to determine coordinates of light of a distortion-free and 
predetermined shape that is projected into a projection area by sensing. 

[0030] 

In other words, the coordinates of the light of the predetermined shape can be 
taken as coordinates of a distortion-free image, and thus a distortion-free image can be 
projected during image projection by mapping the coordinates of an input image into 
those coordinates. 
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[0031] 

This makes it possible to correct keystone distortion in the horizontal direction 
both automatically and appropriately. 
[0032] 

In the image processing system and the projector, the predetermined shape may 
be rectangular or square; 

the area extraction means may extract coordinates of four corners of the light of 
the predetermined shape; and 

the keystone distortion correction means may convert the coordinates of the four 
corners of the light of the predetermined shape in the sensed area into coordinates for 
projection in a spatial light modulator of the projection section, and correct keystone 
distortion by mapping the coordinates of an input image into an area formed by the 
projection coordinates of the four corners. 
[0033] 

In the image processing method, the predetermined shape may be rectangular or 
square; and the method may include: 

extracting coordinates of four corners of the light of the predetermined shape 
when coordinates of the light of the predetermined shape in the sensed area are 
extracted; and 

converting the coordinates of the four corners of the light of the predetermined 
shape in the sensed area into coordinates for projection in the spatial light modulator of 
the projection section, and correcting keystone distortion by mapping coordinates of an 
input image into an area formed by the projection coordinates of the four corners, when 
the keystone distortion is corrected. 

[0034] 

This makes it possible for the image processing system and others to display an 
image of a rectangular or square shape during the display of an input image, by 
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projecting light having a rectangular or square shape, converting coordinates of the four 
corners of that rectangle or square into four coordinates for projection in a spatial light 
modulator (such as a liquid crystal light valve), and mapping the coordinates of an input 
image into an area formed by the four coordinates for projection. 
[0035] 

This enables the image processing system and others to correct keystone 
distortion both automatically and appropriately. 
[0036] 

[Embodiments of the Invention] 

The present invention is described below, taking as an example the application 
thereof to an image processing system that correct keystone distortion of a projected 
image by using a projector that is an image display device and a remote controller that 
is a portable device, with reference to the accompanying figures. Note that the 
embodiments described below do not in any way limit the scope of the present 
invention as laid out in the claims herein. In addition, the entirety of .the configuration 
described with reference to these embodiments is not limited to being essential 
structural components of the present invention. 
[0037] 

The description below concerns a first embodiment in which a remote controller 
having a sensing means is used to sense a projection area and determine a direction, and 
a projector corrects keystone distortion. It also concerns a second embodiment in which 
a remote controller having a light projection means projects a rectangular laser beam 
into a projection area, and a projector corrects keystone distortion by sensing the 
projection area from the projection position. 

[0038] 
First Embodiment 
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An overall conceptual view of an image processing system in accordance with 
the first embodiment is shown in Fig. 1. A schematic view of a projected image 70 and 
histograms in accordance with the first embodiment are shown in Fig. 2. 

[0039] 

5 In the first embodiment, a projector 10 that is a type of image display device 

projects an image from a position which is not directly in front of a projection area 30, 
but directed forward and to the left from a state directly facing the projection area 30, 
and which is also at a distance further to the right than a viewer 40 who is a user. 
[0040] 

10 In such a case, the projected image 70 that the viewer 40 sees has a trapezoidal 

shape such that the leftmost edge is the longest and the rightmost edge is the shortest, as 
shown in Fig. 2. 
[0041] 

With this embodiment, an imaging function is provided in a remote controller 20 
15 that is a type of portable device manipulated by the viewer 40, enabling the remote 
controller 20 to sense (form an image of) the projection area 30. 
[0042] 

The remote controller 20 is also provided with a histogram processing function, 
enabling the remote controller 20 to generate a histogram 80 of the numbers of pixels in 
20 the horizontal direction and a histogram 82 of the numbers of pixels in the vertical 
direction, for the projected image 70 in a captured image 60. 
[0043] 

The remote controller 20 is further provided with a direction determination 
function, enabling the remote controller 20 to generate direction information based on 
25 an angle 0 between a normal direction 0n for the orthogonal direction from a virtual 
viewpoint position 50 to the projection area 30 and a projected image direction 9i from 
the virtual viewpoint position 50 to a central portion of the projected image. 
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[0044] 

Note that the virtual viewpoint position 50 is a position that sets the viewpoint of 
the viewer 40 in a virtual manner, by way of example. In the first embodiment, the 
sensing position of the remote controller 20 becomes the virtual viewpoint position 50. 

[0045] 

The projector 10 also has a keystone distortion correction function that corrects 
any keystone distortion of the projected image 70, based on the histogram information 
(the histogram 80 of the numbers of pixels in the horizontal direction and the histogram 
82 of the numbers of pixels in the vertical direction) and the direction information. 

[0046] 

The image processing system of this embodiment uses these functions to correct 
any keystone distortion of the projected image 70 from consideration of the virtual 
viewpoint position 50, both automatically and appropriately. 

[0047] 

The description now turns to the functional blocks of the projector 10 for 
implementing these functions. 
[0048] 

A functional block diagram of the projector 10 in accordance with this 
embodiment is shown in Fig. 3. 
[0049] 

The projector 10 comprises an input signal processing section 110, a color 
conversion section 120, an output signal processing section 130, an image projection 
section 190, a keystone distortion correction section 140, and a receiver section 160. 

[0050] 

The input signal processing section 110 converts an Rl signal, a Gl signal, and 
a Bl signal (which form RGB signals in analog form that is a type of input image 



information that is input from a personal computer (PC) or the like) into an R2 signal, a 
G2 signal, and a B2 signal in digital form. 
[0051] 

The input signal processing section 110 comprises an A/D converter section 112 
that performs this analog-to-digital conversion and a resizing section 114 that functions 
as part of a keystone distortion correction means to adjust the position and dimensions 
of the projected image 70 by adjusting the input image signal. 

[0052] 

A calibration signal generation section 150 generates an R2 signal, a G2 signal, 
and a B2 signal in digital format that are used for displaying a calibration image. 
[0053] 

Generating the calibration signals within the liquid-crystal projector in this 
manner makes it possible for the liquid-crystal projector itself to perform the calibration, 
without having to input calibration signals to the liquid-crystal projector from an 
external input device such as a PC. Note that the configuration could also be such that 
the calibration signal generation section 150 is not provided and the calibration image 
signals are input from a PC or the like. 
[0054] 

The color conversion section 120 outputs an R3 signal, a G3 signal, and a B3 
signal having a color temperature that has been corrected on the basis of a standard 
setting for the liquid-crystal projector, based on the R2 signal, G2 signal, and B2 signal 
from the input signal processing section 1 10 or the calibration signal generation section 
150. 

[0055] 

The output signal processing section 130 comprises a D/A converter section 132. 
The D/A converter section 132 converts the R3 signal, G3 signal, and B3 signal from 
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the color conversion section 120 into an R4 signal, a G4 signal, and a B4 signal in 
analog format. 
[0056] 

Note that the A/D converter section 112 and the D/A converter section 132 are 
not necessary if the projector 10 uses only RGB signals in digital format. 
[0057] 

The image projection section 190 that functions as projection means comprises a 
spatial light modulator 192, a drive section 194 for driving the spatial light modulator 
192, a light source 196, and a lens 198. 

[0058] 

The drive section 194 drives the spatial light modulator 192, based on the R4 
signal, G4 signal, and B4 signal from the output signal processing section 130. The 
image projection section 190 projects light from the light source 196 through the spatial 
light modulator 192 and the lens 198. 

[0059] 

The receiver section 160 receives histogram information and direction 
information that is transmitted from the remote controller 20. 
[0060] 

The keystone distortion correction section 140 comprises a vertical keystone 
distortion correction section 141, which has an inclination sensor 142 and which 
corrects keystone distortion in the vertical direction, and a horizontal keystone 
distortion correction section 143, which has a 3D-LUT 144 and which corrects keystone 
distortion in the horizontal direction. 

[0061] 

The inclination sensor 142 detects any inclination in the vertical direction of the 
projector 10, and the vertical keystone distortion correction section 141 transmits a 
vertical keystone distortion correction value that corresponds to that inclination (such as 
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an integer value from -n to n, similar to that for manual adjustment) to the resizing 
section 114. 

[0062] 

The horizontal keystone distortion correction section 143 takes the histogram 
information and direction information that has been received by the receiver section 160, 
selects a horizontal direction keystone distortion correction stored in the 3D-LUT 144 
corresponding thereto (such as an integer value from -n to n, similar to that for manual 
adjustment), and transmits the thus-selected horizontal direction keystone distortion 
correction value to the resizing section 114. 

[0063] 

The description now turns to the functional blocks for implementing the 
functions of the remote controller 20. 
[0064] 

A functional block diagram of the remote controller 20 in accordance with this 
embodiment is shown in Fig. 4. 
[0065] 

The remote controller 20 comprises an area sensor section 210 that functions as 
sensing means to obtain the captured image 60 by using a sensor such as a CCD sensor 
to image the projection area 30, a projected image area extraction section 220 that 
extracts the projected image 70 from the captured image 60, and a histogram processing 
section 230 that functions as histogram generation means for generating the histogram 
80 of the numbers of pixels in the horizontal direction and the histogram 82 of the 
numbers of pixels in the vertical direction of the projected image 70. 
[0066] 

The projected image area extraction section 220 extracts a brightness detection 
area of pixels of a maximum brightness, such as those of greater than 65% or 80% 
brightness, from the captured image 60 as the projected image 70. 



21 



[0067] 

The histogram processing section 230 generates the totals of the numbers of 
pixels for each scan line, when the projected image 70 is scanned in the horizontal 
direction, as the histogram 80 of the numbers of pixels in the horizontal direction and 
the totals of the numbers of pixels for each scan line, when the projected image 70 is 
scanned in the vertical direction as the histogram 82 of the numbers of pixels in the 
vertical direction, as shown in Fig. 2. 
[0068] 

The histogram processing section 230 also calculates the ratio of an upper-base 
length a to a lower-base length b from the histogram 80 of the numbers of pixels in the 
horizontal direction and the sign of a value obtained by subtracting a lower-base length 
d from an upper-base length c of the histogram 82 of the numbers of pixels in the 
vertical direction, and transmits them as histogram information to a transmitter section 
250. 

[0069] 

In this case, the ratio of the upper-base length a to the lower-base length b of the 
histogram 80 of the numbers of pixels in the horizontal direction represents the 
magnitude of the distortion. The magnitude of the distortion increases as this ratio 
increases, by way of example. The sign of the value obtained by subtracting the lower- 
base length d from the upper-base length c of the histogram 82 of the numbers of pixels 
in the vertical direction represents the direction of the distortion in either the right or left 
direction. 

[0070] 

The histogram information can therefore be termed information that represents 
kevstone distortion as numerical values. 
[0071] 



The remote controller 20 comprises a direction determination section 240 having 
a direction sensor 242, and the transmitter section 250 that transmits histogram 
information comprising the histogram 80 of the numbers of pixels in the horizontal 
direction and the histogram 82 of the numbers of pixels in the vertical direction from the 
5 histogram processing section 230 and direction information from the direction 
determination section 240, to the projector 10. 
[0072] 

The direction determination section 240 uses the direction sensor 242 to measure 
the normal direction 6n of the optical axis in the direction perpendicular to the 

10 projection area 30 from the virtual viewpoint position 50 and the projected image 
direction Gi of a central portion of the projected image 70 from the virtual, viewpoint 
position 50, as shown in Fig. 1, and generates direction information that represents an 
ansle 6 obtained by subtracting the projected image direction Gi from the normal 
direction 0n to obtain an angle 8. 

15 [0073] 

The direction information represents the positional relationship between the 
virtual viewpoint position 50 and the projector 10. If the angle 9 is zero, by way of 
example, the projector 10 is directly in front of the viewer 40, a positive value of angle 
9 indicates that the projector 10 is to the right of the viewer 40, and a negative value of 
20 angle 9 indicates that the projector 1 0 is to the left of the viewer 40. 
[0074] 

The description now turns to the flow of image processing using the above 
components, with reference to a flowchart. 
[0075] 

?5 A flowchart of the flow of keystone distortion correction in accordance with this 

embodiment is shown in Fig. 5. 
[0076] 
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First of all, the vertical keystone distortion correction section 141 transmits a 
vertical keystone distortion correction value corresponding to the inclination of the 
projector 10 in the vertical direction, as detected by the inclination sensor 142, to the 
resizing section 114. The resizing section 114 resizes the input image signal to perform 
a keystone distortion correction in the vertical direction (step SI). 

[0077] 

The calibration signal generation section 150 generates a calibration signal that 
causes the display of a completely white maximum grayscale image, and the image 
projection section 190 projects that (step S2). 

[0078] 

The viewer 40 points the remote controller 20 in the direction perpendicular to 
the projection area 30 and presses a normal direction measurement button on the remote 
controller 20. In accordance with this action, the direction determination section 240 
measures the normal direction 6n (step S3). 

[0079] 

The viewer 40 then points the remote controller 20 toward the vicinity of the 
center of the projected image 70, which maintaining the horizontal orientation thereof, 
and presses a keystone distortion correction button. In accordance with this action, the 
direction determination section 240 measures the projected image direction Gi (step S4). 

[0080] 

Concomitant with these actions, the area sensor section 210 senses the 
projection area 30 and extracts the area of the projected image 70 from the captured 
image 60 (step S5). 

[0081] 

The histogram processing section 230 generates the histogram 80 of the numbers 
of pixels in the horizontal direction and the histogram 82 of the numbers of pixels in the 
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vertical direction, based on the projected image 70, derives the above-described a ; b, c, 
and d values, and generates the histogram information (step S6). 
[0082] 

The direction determination section 240 derives the angle 6 - On - 9i, and 
generates the direction information (step S7). 
[0083] 

The transmitter section 250 of the remote controller 20 transmits the histogram 
information and direction information to the projector 10. The receiver section 160 of 
the projector 10 receives the histogram information and direction information from the 
remote controller 20. 

[0084] 

The horizontal keystone distortion correction section 143 sends a horizontal 
direction keystone distortion correction value, that is based on a correction value from 
the 3D-LUT 144 corresponding to the histogram information and direction information, 
to the resizing section 1 14. The resizing section 114 corrects the keystone distortion in 
the horizontal direction by correcting the input image signal in such a manner that the 
keystone distortion in the horizontal direction is corrected thereby, based on that 
correction value (step S8). 

[0085] 

The above-described procedure makes it possible for the image processing 
system that comprises the remote controller 20 to correct keystone distortion from 
consideration of the virtual viewpoint position 50, automatically and also appropriately. 

[0086] 

Note that the hardware described below could be employed as the hardware of 
the above-described projector 10. 
[0087] 
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A block diagram of the hardware of the projector 10 in accordance with this 
embodiment is shown in Fig. 6. 
[0088] 

For example, the configuration could be implemented by an A/D converter 930 
5 or the like as the A/D convener section 1 12: an image processing circuit 970, RAM 950, 
and a CPU 910 or the like as the color conversion section 120 and the keystone 
distortion correction section 140; a D/A converter 940 or the like as the D/A converter 
section 132: and a liquid-crystal panel 920 and a ROM 960 in which is stored liquid- 
crystal light valve drivers for driving the liquid-crystal panel 920 as the spatial light 
10 modulator 192. 

[0089] 

Note that these components can exchange information between themselves over 
a system bus 980. 
[0090] 

15 The transfer paths between the projector 10 and the remote controller 20 could 

be either cables or wireless. Infrared ports or the like could be used for wireless 
information exchange or input-output ports or the like could be used for wired 
information exchange, between the receiver section 160 of the projector 10 and the 
transmitter section 250 of the remote controller 20. 

20 [0091] 

The rest of the hardware of the remote controller 20 could be implemented by 
using a CCD sensor or the like as the area sensor section 210 and image processing 
circuitry as the projected image area extraction section 220 and the histogram 
processing section 230. 
25 [0092] 

In addition, these components could be implemented in a hardware manner by 
circuitry or in a software manner by drivers. 
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[0093] 

The keystone distortion correction section 140 could also be implemented by 
reading from an information storage medium 900 a program that causes a computer to 
function as the keystone distortion correction section 140, to cause the computer to 
function as the keystone distortion correction section 140. 

[0094] 

The information storage medium 900 could be a CD-ROM, DVD-ROM, ROM, 
RAM, or HDD, by way of example, and the method of reading the program therefrom 
could be a direct method or an indirect method. 

[0095] 

Instead of the information storage medium 900, it is also possible to download a 
program that implements the above-described functions, from a host device through a 
transfer path, in order to implement the above-described functions. 

[0096] 

As described above, this embodiment makes it possible for the image processing 
system to determine the keystone distortion by sensing from the virtual viewpoint 
position 50 which is the sensing position. The image processing system can convert the 
magnitude of the distortion into numerical values by converting the sensing information 
into the histogram 80 of the numbers of pixels in the horizontal direction and the 
histogram 82 of the numbers of pixels in the vertical direction. In addition, the image 
processing system can determine any relative displacement between the sensing 
direction and the projection direction by determining the angle 9. 
[0097] 

The image processing system can also correct not only keystone distortion in the 
vertical direction, but also keystone distortion in the horizontal direction, both 
automatically and appropriately, by correcting the keystone distortion based on 
histogram information that represents the distortion circumstances and direction 
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information that represents the relative displacement between the sensing direction and 
the projection direction. 
[0098] 

The keystone distortion in the horizontal direction varies with the relative 
positional relationship between the projector 10 and the projection area 30 ; whereas the 
keystone distortion in the vertical direction is mainly affected by the inclination of the 
projector 10. 

[0099] 

Since the image processing system can therefore use the inclination sensor 142 
provided in the projector 10 to correct keystone distortion in the vertical direction and 
also use the histogram information that represents the relative positional relationship 
between the projector 10 and the projection area 30 to correct keystone distortion in the 
horizontal direction, it can correct keystone distortion in both the vertical direction and 
the horizontal direction as appropriate. 

[0100] 
Second embodiment 

The description now turns to a second embodiment that corrects keystone 
distortion by projecting a rectangular laser beam from the virtual viewpoint position 50 
to the projection area 30 and sensing the projection area 30 from a projection position 
52. Note that the virtual viewpoint position 50 in this second embodiment is the position 
from which the laser beam is projected. 

[0101] 

A schematic view of the projected image 70 and a laser beam projected image 
90 in accordance with the second embodiment is shown in Fig. 7. 
[0102] 
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In this embodiment, a laser beam is projected from the remote controller 
operated by the viewer 40 within the projected image 70, in a rectangular form as seen 
by the viewer 40. 

[0103] 

As a result, the captured image 60 comprises the projected image 70 that is a 
trapezium ABCD and the laser beam projected image 90 that is a rectangle EFGH 
within the trapezium ABCD. Note that the laser beam projected image 90 is preferably 
formed within the projected image 70. 

[0104] 

The projected image 70 is reshaped into a rectangle to correct keystone 
distortion, by associating the coordinates of the four corners of the rectangle EFGH with 
four conversion coordinates within the spatial light modulator 192 then mapping the 
coordinates of the input image into those four conversion coordinates. 

[0105] 

The description now turns to the functional blocks of the projector 10 and the 
remote controller for implementing these functions. 
[0106] 

A functional block diagram of the projector 10 in accordance with this 
embodiment is shown in Fig. 8. A functional block diagram of a remote controller 21 in 
accordance with this embodiment is shown in Fig. 9. 

[0107] 

The remote controller 21 of this embodiment comprises a laser beam projection 
section 280 that functions as a light projection means, an operating section 270, and a 
transmitter section 260 that transmits signals such as an operation- start signal to the 
projector 10. 

[0108] 
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The projector 10 of this embodiment has a similar configuration to that of the 
projector 10 of Fig. 3, described as the first embodiment, except that it differs from the 
projector 10 of Fig. 3 in the functions of the keystone distortion correction section 140 
and in the provision of an area sensor section 180 that functions as sensing means. 

[0109] 

The keystone distortion correction section 140 of this embodiment comprises a 
control section 145, a determination section 146, a coordinate conversion section 147, 
and an area extraction section 148 that extracts the coordinates of the four corners of the 
projected image 70 in the captured image 60 and the coordinates of the four corners of 
the laser beam projected image 90 in the captured image 60, based on sensing 
information. 

[0110] 

The area sensor section 180 is provided with the projector 10 of Fig. 1, for 
sensing the projection area 30 from the projection position 52. 
[0111] 

Note that the brightness and color of the projected image 70 differ from the 
brightness and color of the laser beam projected image 90. It is therefore possible for the 
area extraction section 148 to distinguish between the laser beam projected image 90 
and the projected image 70 within the captured image 60, by demarcating in accordance 
with brightness or color. 
[0112] 

The description now turns to the flow of processing with these components. 
[0113] 

A flowchart of the flow of processing by the remote controller 20 in accordance 
with this embodiment is shown in Fig. 10. 
[0114] 



First of all, the description relates to the flow of processing by the remote 
controller 20. 
[0115] 

The viewer faces the projection area 30 and presses a distortion correction 
button of the remote controller 21. This makes the operating section 270 output a 
control signal to the transmitter section 260, and the transmitter section 260 transmits a 
start signal (meaning start the keystone distortion correction) to the projector 10 (step 
Sll). 

[0116] 

The viewer 40 presses a projection button of the remote controller 21 when a 
patch (calibration image) is projected by the projector 10 (step S12). This makes the 
operating section 270 output a control signal to the laser beam projection section 280 to 
make it output a laser beam. 

[0117] 

The laser beam projection section 280 projects a rectangular laser beam based on 
the control signal (step SI 3). 
[0118] 

The description now concerns the flow of processing by the projector 10. 
[0119] 

A flowchart of the flow of processing by the projector 10 in accordance with this 
embodiment is shown in Fig. 1 1 . 
[0120] 

The determination section 146 determines whether or not the receiver section 
160 has received a keystone distortion correction start signal from the remote controller 
21 (step S21). 

[0121] 
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If it has received the start signal, the image projection section 190 projects a ' 
completely-white patch, based on a calibration signal by the calibration signal 
generation section 150 (step S22). 

[0122] 

5 Note that when the patch is projected, the viewer 40 adjusts the position and 

direction of the remote controller 21 in such a manner that the laser beam is displayed 
as a rectangle within the projected image 70 of the patch. 
[0123] 

The area sensor section 180 senses the projection area 30 in which the laser 
10 beam projected image 90 and the projected image 70 are displayed, and generates 
sensing information (step S23). 
[0124] 

The area extraction section 148 extracts the projected image area (the 
coordinates of the sensed area defined by the four corners ABCD of the projected image 
15 70) that has brightnesses within the range of the projected light, based on the sensing 
information (step S24). 
[0125] 

Concomitant therewith, the area extraction section 148 extracts the laser frame 
area (the coordinates of the sensed area defined by the four corners EFGH of the laser 
20 beam projected image 90) that has brightnesses within the range of the laser beam, 
based on the sensing information (step S25). 
[0126] 

The determination section 146 determines whether or not the laser frame area is 
within the projected image area, based on the coordinates extracted by the area 
25 extraction section 148 (step S26). 
[0127] 
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If the laser frame area is within the projected image area, the coordinate 
conversion section 147 converts the coordinates EFGH of the laser frame area into 
coordinates E'F'G'H' in the processing system of the spatial light modulator 192 of the 
image projection section 190 (step S27). 

[0128] 

The resizing section 114 maps the input image signal into an area formed by the 
coordinates E'F'G'H'. 
[0129] 

The image projection section 190 projects an image based on the thus-mapped 
image information (step S29). 
[0130] 

Note that if the laser frame area is not within the projected image area, the 
control section 145 outputs a control signal to the calibration signal generation section 
150. so as to cause the projection of a message image prompting the viewer 40 to 
project the laser frame area into the projected image area. 
[0131] 

The calibration signal generation section 150 generates the signal for that 
message image and the image projection section 190 displays the message based on the 
signal for that message image (step S28). 

[0132] 

In this manner, this embodiment makes it possible to determine the coordinates 
of a rectangle of light created by light of a rectangular shape projected into the 
projection area 30 in a state in which there is no distortion when seen from the virtual 
viewpoint position 50, by sensing from the projection position 52. 

[0133] 

In other words, the coordinates of the rectangle of light enable the capture of the 
coordinates of a distortion-free image, and a distortion-free image can be projected 



during the projection of images by mapping the coordinates of the input image onto 
those coordinates. 
[0134] 

This makes it possible to correct keystone distortion not only in the vertical 
direction but also in the horizontal direction, both automatically and appropriately. 
[0135] 

In comparison with the first embodiment there is no need for the 3D-LUT 144, 
enabling a reduction in necessary storage capacity. 

[0136] 
Modifications 

The present invention was described above with reference to preferred 
embodiments thereof but it should be obvious to those skilled in the art that the 
application of the present invention is not limited to the above-described embodiments. 

[0137] 

For example, an input image signal is mapped into a laser frame area in the 
second embodiment, but it is equally possible to use laser beam information only in the 
obtaining of distortion information. As an example thereof, it is possible to obtain an 
imaee such that keystone distortion is corrected within suitable dimensions (such as the 
largest possible dimensions), irrespective of the dimensions of the laser frame, by using 
the system of the second embodiment to project and sense a laser frame, determining 
distortion of the laser frame of the sensed image by the histogram processing section of 
the first embodiment (in this case, within the projector 10), and resizing the input image 
in accordance with the distortion circumstances. 
[0138] 

; Furthermore, the viewer 40 is assumed to be placed facing the projection area 30 

in the second embodiment, but the configuration could be such that the direction 
determination section 240 of the remote controller 20 of the first embodiment is 
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provided in the remote controller 21 of the second embodiment, and the histogram 
processing section of the remote controller 20 of the first embodiment and the 3D-LUT 
of the projector 10 of the first embodiment are provided in the projector 10 of the 
second embodiment. In addition, the necessity of positioning the viewer 40 facing the 
5 projection area 30 can be removed by having the projector 10 determine the direction 
and laser beam distortion information, to perform keystone distortion correction. 
[0139] 

In the second embodiment, the laser beam was described as seeming to be 
rectangular, by way of example, but it could equally well be another shape such as 
10 square or circular. 

[0140] - 

Furthermore, a laser beam was used in the second embodiment, but any light 
that can be distinguished from the projected image 70 can be used therefore, such as 
infrared light. 
15 [0141] 

In the first embodiment, the inclination sensor 142 is used in the correction of 
the keystone distortion in the vertical direction, but keystone distortion in both the 
vertical and horizontal directions could be corrected on the basis of the histogram 
information and direction information alone, without using the inclination sensor 142. 
20 [0142] 

In addition, if the lens 198 and the area sensor section 210 are provided with a 
zoom function, the projector. 10 could correct keystone distortion by obtaining zoom- 
related information (such as numerical values that express the state at the furthest 
distance as 0 and the state at the widest angle as 1). 
25 [0143] 

This enables automatic and also appropriate keystone distortion correction when 
using either the long-distance function or the wide-angle function. 
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[0144] 

In the first embodiment, the remote controller is used to measure the projected 
im age direction 0i in the vicinity of the center of the projected image 70 and sense the 
projection area. 30, in the processing of steps S4 and S5, but the following processing 
could be used to ensure that the remote controller is directed reliably toward the center 
of the projected image 70. 
[0145] 

In other words, the configuration could be such that the projector 10 displays a 
predetermined mark (such as an "X") at the center of the projected image 70, the user 
uses a laser pointing function of the remote controller 20 to press the keystone distortion 
correction button while superimposing a laser beam on that predetermined mark, and 
the remote controller 20 measures the projected image direction Gi and senses the 
projection area 30. This makes it possible for the image processing system to direct the 
remote controller more accurately towards the vicinity of the center of the projected 
image 70. 

[0146] 

In the above-described embodiments, the calibration patch was described as 
being a completely-white image, but it is not limited to being completely white and thus 
various other calibration images could be used therefore. 

[0147] 

The configuration of the image processing system and the distribution of 
functions between the projector 10 and the remote controllers 20 and 21 are also not 
limited to those of the above-described embodiments. The functions of the projected 
image area extraction section 220 and the histogram processing section 230 of the first 
S embodiment could be provided in the projector 10 of the first embodiment, by way of 
example. Similarly, the function of the horizontal keystone distortion correction section 
143 of the projector 10 of the first embodiment could be provided in the remote 
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controller 20. Furthermore, the function of the keystone distortion correction section 
140 of the second embodiment could be provided in the remote controller 21 . 
[0148] 

In addition, the virtual viewpoint position 50 and the projection position 52 are 
not limited to those shown in Fig. 1 . 
[0149] 

In the above-described embodiments, the remote controllers 20 and 21 were 
used as portable devices, but other portable devices such as handy terminals, PDAs, or 
mobile phones could equally well be used therefore. 

[0150] 

The embodiments described above related to a liquid-crystal projector as the 
image display device, but they can also be applied to a projector using a digital 
micromirror device (DMD), by way of example. Note that DMD is a trademark 
registered to Texas Instruments Inc. of the USA. 
[0151] 

Note that the functions of the above-described projector 10 could be 
implemented by the projector alone, by way of example, or they could be implemented 
by distributing them between a plurality of processing devices (such as between the 
projector and a PC). 

[Brief Description of the Drawings] 
[Fig. 1] 

Fie. 1 is an overall conceptual view of an image processing system in 
accordance with a first embodiment of the present invention; 
[Fig. 2] 

Fig. 2 is a schematic view of a projected image and histograms in accordance 
with the first embodiment of the present invention; 



[Fig. 3] 

Fig. 3 is a functional block diagram of a projector in accordance with the first 
embodiment of the present invention; 
[Fig. 4] 

5 Fie. 4 is a functional block diagram of a remote controller in accordance with 

the first embodiment of the present invention; 
[Fig. 5] 

Fia. 5 is a flowchart of keystone distortion correction processing in accordance 

with the first embodiment of the present invention; 

10 [Fig- 6] 

Fie. 6 is a hardware block diagram of the projector in accordance with the first 

embodiment of the present invention: 

[Fig. 7] 

Fig. 7 is a schematic view of a projected image and a laser beam projected image 
1 5 in accordance with a second embodiment of the present invention: 
[Fig. 8] 

Fig. 8 is a functional block diagram of a projector in accordance with the second 
embodiment of the present invention; 
[Fig. 9] 

20 Fig. 9 is a functional block diagram of a remote controller in accordance with 

the second embodiment of the present invention; 
[Fig. 10] 

Fig. 10 is a flowchart of the processing by the remote controller in accordance 

with the second embodiment of the present invention; and 

25 [Fig. 11] 

Fig. 1 1 is a flowchart of the processing by the projector in accordance with the 

second embodiment of the present invention. 
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[Description of Reference Numerals] 

10 Projector (Image Display Device) 

20 Remote Controller (Portable Device) 

30 Projection Area 

40 Viewer 

50 Virtual Viewpoint Position (Sensing Position) 

60 Captured Image 

70 Projected Image 

90 Laser Beam Projected Image 

1 1 4 Resizing Section (Keystone Distortion Correction Means) 

1 40 Keystone Distortion Correction Section (Keystone Distortion Correction Means) 

1 4 1 Vertical Keystone Distortion Correction Section 

1 43 Horizontal Keystone Distortion Correction Section 

1 48 Area Extraction Section 

1 60 Receiver Section 

1 80 and 2 1 0 Area Sensor Section (Sensing Means) 

1 90 Image Projection Section (Projection Means) 

192 Spatial Light Modulator 

220 Projected Image Area Extraction Section (Histogram Generation Means) 

230 Histogram Processing Section (Histogram Generation Means) 

240 Direction Determination Section 

260 Transmitter Section 

280 Laser Beam Projection Section (Light Projection Means) 



[Title of Document] ABSTRACT 
[Abstract] 

[Object] An image processing system, a projector, a portable device, and an image 
processing method that enable the correction of keystone distortion in the horizontal 
direction automatically and also appropriately are provided. 

[Arrangement] A remote controller operated by the viewer is provided with a projection 
area imaging function and a direction determination function: a receiver section 160 is 
used to receive imaging information and direction information; a horizontal keystone 
distortion correction section 143 is used to select a keystone distortion correction value 
from a 3D-LUT 144 containing keystone distortion correction values, based on the 
imaging information and direction information; and a resizing section 114 is used to 
correct the keystone distortion. 
[Selected Figure] Fig. 3 
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